Combining different type coercion components for deferred type evaluation

ABSTRACT

In a method of answering questions, a question is received, a question LAT is determined, and a candidate answer to the question is identified. Preliminary types for the candidate answer are determined using first components to produce the preliminary types. Each of the first components produces a preliminary type using different methods. A first type-score representing a degree of match between the preliminary type and the question LAT is produced. Each preliminary type and each first type-score is evaluated using second components. Each of the second components produces a second score based on a combination of the first type-score and a measure of degree that the preliminary type matches the question LAT. The second components use different methods to produce the second score. A final score representing a degree of confidence that the candidate answer matches the question LAT is calculated based on the second score.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.13/835,411 filed Mar. 15, 2013, the entire teachings of which areincorporated herein by reference, in their entirety.

BACKGROUND

The present disclosure relates to question answering (QA) systems, andmore specifically, to determining if a candidate answer is of anappropriate lexical answer type.

The concept of deferred type evaluation is explained in U.S. patentapplication Ser. No. 12/126,642 entitled, “System and Method forProviding Question and Answers with Deferred Type Evaluation,” filed May23, 2008, published as U.S. Patent Application Publication No. US2009/0292687 A1, and issued as U.S. Pat. No. 8,332,394 B2, the contentsof which are incorporated herein by reference.

The concept of deferred type evaluation was further discussed inpreviously filed U.S. patent application Ser. No. 13/239,165 entitled,“Providing Question and Answers with Deferred Type Evaluation Using Textwith Limited Structure,” filed Sep. 21, 2011, published as U.S. PatentApplication Publication No. US 2012/0078902 A1, the contents of whichare incorporated herein by reference.

SUMMARY

The systems and methods herein address the coverage problem fromindividual type coercion components. Currently, there is a variety ofways to determine if a candidate answer is of a lexical answer type(LAT). All of these solutions rely on a particular set of resources anda particular set of algorithms. Since each type coercion component hasits own limitations and biases, the coverage of type coercion in generalcan be improved.

Systems and methods herein combine the intermediate outputs of differenttype coercion components to produce a new type coercion output that hasimproved coverage than the other type coercion components, individually.

The present disclosure leverages the concept of LATs. The LATs arecomputed from a natural language analysis of the query and provide adescription of an answer rather than its ontological category. A LAT ofthe question/query is the descriptor of the referent or referents of theentity that is a valid answer to the question.

According to a method herein, a question is received into a computerizeddevice. A question LAT is determined, using the computerized device. Thequestion LAT is a lexical answer type associated with the question. Acandidate answer to the question is identified, using the computerizeddevice. Preliminary types for the candidate answer are automaticallydetermined using first components of the computerized device. The firstcomponents use different methods to produce the preliminary types, andeach of the first components produces a preliminary type. A matchbetween the preliminary type and the question LAT is automaticallyscored, using the computerized device. Each of the first componentsproduces a first type-score. The first type-score represents a degree ofmatch between the preliminary type and the question LAT. The scoring isdifferentiated based on which of the components produced the preliminarytype. Each preliminary type and each first type-score is automaticallyevaluated using second components of the computerized device. Each ofthe second components produces a second score based on a combination ofthe first type-score and a measure of degree that the preliminary typematches the question LAT. The second components use different methods toproduce the second score. A final score is automatically calculatedbased on the second score from each of the second components, using thecomputerized device. The final score, representing a degree ofconfidence that the candidate answer is a type that matches the questionLAT, is automatically output, using the computerized device.

According to a method herein, a lexical answer type (LAT) associatedwith a question in a question-answering system is automaticallyidentified, to produce a question LAT, using a computerized device. Acandidate answer to the question is automatically generated, using thecomputerized device. Preliminary types for the candidate answer areautomatically determined using first components of the computerizeddevice. The first components use different resources to produce thepreliminary types, and each of the first components produces apreliminary type for the candidate answer. A match between thepreliminary type and the question LAT is automatically scored using thefirst components of the computerized device, producing a firsttype-score for each preliminary type based on an amount that thepreliminary type corresponds to the question LAT. The first componentsuse different resources to produce the first type-score for thepreliminary type, and the scoring is differentiated based on which ofthe first components produced the preliminary type. Each of secondcomponents automatically evaluates each preliminary type and the firsttype-score, using the computerized device. Each of the second componentsproduces a second score for the preliminary type for the candidateanswer based on a combination of the first type-score and a measure ofdegree that the preliminary type matches the question LAT. A final scorebased on the second score from each of the second components isautomatically output, using the computerized device. The final scorerepresents a degree of confidence that the candidate answer is a typethat matches the question LAT.

According to a computer system for determining a confidence score forcandidate answers to questions in a question-answering system herein,the system comprises an automated question answering (QA) systemcomprising a query analysis module, a candidate answer generatoroperatively connected to the query analysis module, a processorcomprising a plurality of first components, a plurality of secondcomponents operatively connected to the plurality of first components,and a classifier operatively connected to the plurality of secondcomponents, and a user interface operatively connected to the queryanalysis module. The user interface receives a question into theautomated QA system. The query analysis module determines a questionlexical answer type (LAT) for the question. The candidate answergenerator automatically computes a candidate answer to the question. Theprocessor automatically calculates a preliminary type for the candidateanswer from each of the plurality of first components. The firstcomponents use different methods to produce the preliminary type. Theprocessor automatically scores a match between the preliminary types andthe question LAT to produce a first type-score from each of theplurality of first components based on an amount that the preliminarytype corresponds to the question LAT. The scoring is differentiatedbased on which of the first components produced the preliminary type. Asame preliminary type receives a different type-score based ondifferences between the first components that produced the samecandidate answer type. The processor automatically evaluates eachpreliminary type and first type-score from each of the plurality offirst components using each of the plurality of second components andproduces a second score for the preliminary type based on a measure ofdegree that the preliminary type matches the question LAT for each ofthe plurality of second components. The processor automatically outputsa final score for the preliminary type from the classifier based on thesecond score for the preliminary type from each of the plurality ofsecond components.

According to a computer program product for determining a confidencescore for candidate answers to questions in a question-answering systemherein, the computer program product comprises a tangible computerreadable storage medium having program code embodied therewith. Theprogram code is readable and executable by a computer to perform amethod. According to the method, a question is received into acomputerized device. A question LAT is determined. The question LAT is alexical answer type associated with the question. A candidate answer tothe question is identified. Preliminary types for the candidate answerare automatically determined using first components of the computerizeddevice. The first components use different methods to produce thepreliminary types, and each of the first components produces apreliminary type. A match between the preliminary type and the questionLAT is automatically scored. Each of the first components produces afirst type-score. The first type-score represents a degree of matchbetween the preliminary type and the question LAT. The scoring isdifferentiated based on which of the components produced the preliminarytype. Each preliminary type and each first type-score is automaticallyevaluated using second components of the computerized device. Each ofthe second components produces a second score based on a combination ofthe first type-score and a measure of degree that the preliminary typematches the question LAT. The second components use different methods toproduce the second score. A final score is automatically calculatedbased on the second score from each of the second components. The finalscore, representing a degree of confidence that the candidate answer isa type that matches the question LAT, is automatically output.

BRIEF DESCRIPTION OF THE DRAWINGS

The systems and methods herein will be better understood from thefollowing detailed description with reference to the drawings, which arenot necessarily drawn to scale and in which:

FIG. 1 is a high-level system diagram according to systems and methodsherein;

FIG. 2 is a block diagram illustrating various aspects of systems andmethods herein;

FIG. 3 is a block diagram illustrating various aspects of systems andmethods herein;

FIG. 4 is a block diagram illustrating various aspects of systems andmethods herein;

FIG. 5 is a flow diagram illustrating systems and methods herein;

FIG. 6 is a schematic diagram of a hardware system according to systemsand methods herein;

FIG. 7 is a schematic diagram of a deployment system according tosystems and methods herein;

FIG. 8 is a schematic diagram of an integration system according tosystems and methods herein;

FIG. 9 is a schematic diagram of an on demand system according tosystems and methods herein;

FIG. 10 is a schematic diagram of a virtual private network systemaccording to systems and methods herein; and

FIG. 11 is a schematic diagram of a virtual private network systemaccording to systems and methods herein.

DETAILED DESCRIPTION

It will be readily understood that the systems and methods of thepresent disclosure, as generally described and illustrated in thedrawings herein, may be arranged and designed in a wide variety ofdifferent configurations in addition to the systems and methodsdescribed herein. Thus, the following detailed description of thesystems and methods, as represented in the drawings, is not intended tolimit the scope defined by the appended claims, but is merelyrepresentative of selected systems and methods. The followingdescription is intended only by way of example, and simply illustratescertain concepts of the systems and methods, as disclosed and claimedherein.

As will be referred to herein, the word “question” and “query”, andtheir extensions, are used interchangeably and refer to the sameconcept, namely a request for information. Such requests are typicallyexpressed in an interrogative sentence, but they can also be expressedin other forms, for example as a declarative sentence providing adescription of an entity of interest (where the request for theidentification of the entity can be inferred from the context).“Structured information” (from “structured information sources”) isdefined herein as information whose intended meaning is unambiguous andexplicitly represented in the structure or format of the data (e.g., adatabase table). “Unstructured information” (from “unstructuredinformation sources”) is defined herein as information whose intendedmeaning is only implied by its content (e.g., a natural languagedocument). By “Semi-structured”, it is meant data having some of themeaning explicitly represented in the format of the data, for example aportion of the document can be tagged as a “title”.

FIG. 1 shows a high-level system diagram depicting logical architecturefor a system 100 described herein. As shown in FIG. 1, the system 100includes a Query Analysis module 112 implementing functions forreceiving and analyzing a user question or query 118. According to onesystem and method, a “user” 115 refers to a person or personsinteracting with the system 100, and the term “user query” refers to aquery 118 (and its context) posed by a user 115. However, it isunderstood that other systems and methods can be constructed, where theterm “user” refers to a computer system 120 generating a query 118 bymechanical means, and where the term “user query” refers to such amechanically generated query and context. A Candidate Answer Generationmodule 123 is provided to implement a search for candidate answers bytraversing Primary Sources 126 having structured, semi structured, andunstructured sources and an Answer Source Knowledge Base 129 containingcollections of relations and lists extracted from primary sources. Allthe sources of information can be locally stored or distributed over anetwork, including the Internet. The Candidate Answer Generation module123 generates a plurality of output data structures containing candidateanswers 132 based upon the analysis of retrieved data.

In FIG. 1, a first system and method is depicted that includes anEvidence Gathering and Answer Scoring module 135 that interfaces withthe Primary Sources 126 and the Answer Source Knowledge Base 129 toconcurrently analyze the evidence based on passages having candidateanswers and score each candidate answer 132, as parallel processingoperations. The Evidence Gathering and Answer Scoring module 135comprises a Candidate Answer Scoring module 138 for analyzing aretrieved passage and scoring each of the candidate answers 132 of aretrieved passage.

The Answer Source Knowledge Base 129 may comprise a corpus of datacomprising one or more databases of structured or semi-structuredsources (pre-computed or otherwise) comprising collections of relations(e.g., Typed Lists). In an exemplary implementation, the Answer SourceKnowledge Base 129 may comprise a database stored in a memory storagesystem, e.g., a hard drive. The Answer Source Knowledge Base 129 mayinterface with several structured and unstructured sources includingTyped Lists (e.g., list of all countries in the world), Precise Unary(e.g., a country), Binary (e.g., country+head of state of that country),Ternary (e.g., country+head of state of that country+wife of the head ofstate), n-ary Relation Extracted, etc.

An Answer Ranking module 141 provides functionality for rankingcandidate answers 132 and determining a response 144 returned to a user115 via a user's computer display interface (not shown) or a computersystem 120, where the response may be an answer, an elaboration of aprior answer, or a request for clarification in response to a questionwhen a high quality answer to the question is not found.

FIG. 1 shows a machine learning implementation where the Answer Rankingmodule 141 includes a trained model component 147 produced using amachine learning techniques from prior data. The prior data may encodeinformation on features of candidate answers 132, the features ofpassages the candidate answers come from, the scores given to them byCandidate Answer Scoring modules 138, and whether the candidate answer132 was correct or not. In other words, machine-learning algorithms canbe applied to the entire content of the Candidate Answer Scoring modules138 together with the information about correctness of the candidateanswer. Such prior data is readily available for instance in technicalservices support functions, or in more general setting on the Internet,where many websites list questions with correct answers. The modelencodes a prediction function that is its input to the learned featurecombination module 148 shown in FIG. 1. The Answer Ranking module 141may also include a module to produce a ranked list of answers 149 toprovide a qualitative rank for the response 144 returned to a user 115.

It is understood that skilled artisans may implement a further extensionto the system of the disclosure shown in FIG. 1, to employ one or moremodules for enabling I/O communication between a user or computer systemand the system 100 according to, but not limited to, the followingmodalities of text, audio, video, gesture, tactile input and output,etc. Thus, both an input query and a generated query response may beprovided in accordance with one or more of multiple modalities includingtext, audio, image, video, tactile, or gesture. Thus, for example, if aquestion is posed using other modalities, e.g., a series of imagespointed by the user, the method applies to the textual aspects of theimages, captured in their descriptions or inferred by an analysis system(not shown).

The system 100 depicted in FIG. 1, may be local, on a server, or servercluster, within an enterprise, or alternately, may be distributed withor integral with or otherwise operate in conjunction with a public orprivately available search engine in order to enhance the questionanswer functionality in the manner as described. Thus, the systems andmethods may be provided as a computer program products comprisinginstructions executable by a processing device, or as a servicedeploying the computer program product. The architecture employs asearch engine (a document retrieval system) as a part of CandidateAnswer Generation module 123, which may be dedicated to the Internet, apublicly available database, a website (e.g., IMDB.com) or, a privatelyavailable database. Databases can be stored in any storage system, e.g.,a hard drive or flash memory, and can be distributed over the network.

As shown in FIG. 1, the Query Analysis module 112 receives an inputcomprising the query 118 entered, for example, by a user 115 via aweb-based browser device. A query 118 input to the system 100 maycomprise a string such as “Who was the tallest American president?”.Alternately, a query 118 may consist of a string and an implicitcontext, e.g., “Who was the shortest?”. In this example, context mayrange from a simple string e.g., “American presidents” or “Who was thetallest American president?” to any data structure, e.g., allintermediate results of processing of the previous strings—a situationarising e.g., in a multiple turn dialog. The query 118 is received bythe Query Analysis module 112 which includes, but is not limited to, oneor more of the following sub-processes: A Parse and Predicate ArgumentStructure block (not shown) that implements functions and programminginterfaces for decomposing an input query into its grammatical andsemantic components, e.g., noun phrases, verb phrases, andpredicate/argument structure. An (English Slot Grammar) ESG-type parsermay be used to implement parsing. A Focus Segment, Focus & Modifiersblock may be provided that computes the focus and focus modifiers of thequestion. Further implementations may further include a QuestionDecomposition block (not shown) in the Query Analysis module 112 thatimplements functions and programming interfaces for analyzing the query118 to determine the sets of constraints specified by the question aboutthe target answer. In accordance with systems and methods herein, theQuery Analysis module 112 includes a Lexical Answer Type (LAT) block 152that implements functions and programming interfaces to provideadditional constraints on the answer type. The computation in the QueryAnalysis module 112 comprises, but is not limited to, the Lexical AnswerType (LAT). The LAT is computed from a natural language analysis of thequery 118 and provides a description of an answer rather than itsontological category.

In FIG. 1, the LAT block 152 includes certain functions/sub-functions(not shown) to determine the LAT. These sub-functions include a parsersuch as the ESG parser as described herein above, and a co-referenceresolution module (as described, for example, inhttp://www.isi.edu/.about.hobbs/muc5-generic-final.pdf; andhttp://gate.ac.uk/sale/taln02/taln-ws-coref.pdf).

The certain functions/sub-functions operate to compute a LAT from anatural language analysis of the query and provide a description of ananswer rather than its ontological category. Thus, for example, theitalicized words in the following sentence represent the LAT “Aftercircumnavigating the Earth, which explorer became mayor of Plymouth,England?” the answer must include both “explorer” and “mayor”; and thesetwo strings become the question LATs.

As mentioned above, a LAT of the question/query 118 is the type (i.e.,the descriptor) of the referent of the entity that is a valid answer tothe question. In practice, LAT is the descriptor of the answer detectedby a natural language understanding module (not shown) comprising acollection of patterns or a parser with a semantic interpreter.

With reference to the LAT block 152, in the Query Analysis module 112 ofFIG. 1, the LAT represents the question terms that identify the semantictype of a correct answer. As is known, a LAT may be detected in aquestion through pattern rules such as “any noun phrase that follows thewh-word and serves as the subject or the object of the main verb in aquestion is a LAT”. For example, in the question “Which Dublin-bornactor once married Ellen Barkin?”, the noun phrase “Dublin-born actor”follows the wh-word “which”, and is the subject of the main verb,“marry”. LAT detection rules can be encoded manually or learned bymachine automatically through association rule learning. In this case,the natural language understanding module can be limited toimplementation the simple rules as described above.

LATs should include modifiers of the main noun if they change itsmeaning. For example, a phrase “body of water” has different meaningthan “water” or “body”, and therefore in the following query the LAT hasto include the whole phrase (italicized):

“Joliet and Co. found that the Mississippi emptied into what body ofwater?”

It is understood that multiple LATs can be present in the query and thecontext, and can even be present in the same clause. For example, theitalicized words represent the LAT in the following queries:

“In 1581, a year after circumnavigating the Earth, which explorer becamemayor of Plymouth, England?”

“Which New York City river is actually a tidal strait connecting upperNew York Bay with Long Island Sound?”

Even though in many cases the LAT of the question can be computed usingsimple rules as described herein above, in other situations, such aswhen multiple LATs are present, the LATs are computed based ongrammatical and predicate argument structure. Thus the natural languageunderstanding module should include a parser (such as ESG is used tocompute the grammatical structures) and a shallow semantic interpreterto compute the semantic co-reference between the discourse entities,such as “river” and “tidal strait” or “explorer” and “mayor”, to addboth of them to the list of LATs. It is understood that the LATs caninclude modifiers.

Thus, in the first example above, the list of LATs may contain explorer,mayor, mayor of Plymouth, mayor of Plymouth, England. A minimal possiblenoun phrase that identifies the answer type corresponds to the maximalentity set, and the maximal noun phrase provides the best match.

According to systems and methods herein, the Evidence Gathering andAnswer Scoring module 135 includes several type coercion (TyCor)components 155, as described in further detail below, that test thecandidate answers in order to determine that the candidate answer 132 isan appropriate type for the question/query 118.

Referring to FIG. 2, the type coercion framework consists of a suite ofanswer scoring components that each take a question LAT 207 and acandidate answer 132, and determine if the candidate answer 132 is ofthe lexical type of the question, returning a measure of degree that thetype of the candidate answer is an appropriate lexical type 215. Foreach candidate answer 132, the candidate is matched against instances ina corpus of data (e.g., the Answer Source Knowledge Base 129). Candidatetypes associated with those instances in the corpus of data areretrieved. The question LAT 207 is matched with the candidate types, anda score is produced representing the degree of match. Each of the TyCorcomponents 155 uses a source of typing information and performs severalsteps, such as described below, each of which is capable of error thatimpacts its confidence:

Entity Disambiguation and Matching (EDM):

The most obvious, and most error-prone, step in using an existing sourceof typing information is to find the entity in that source thatcorresponds to the candidate answer. Since the candidate is just astring, this step accounts for both polysemy (the same name may refer tomany entities) and synonymy (the same entity may have multiple names).Each source may require its own special EDM implementations that exploitproperties of the source; for example, DBpedia encodes useful naminginformation in the entity URI. EDM implementations typically try to usesome context for the answer, but in purely structured sources, thiscontext may be difficult to exploit.

Predicate Disambiguation and Matching (PDM):

Similar to EDM, the type in the source that corresponds to the LAT isfound. In some sources, this is the same algorithm as EDM, in others,type looking requires special treatment. In a few, especially thoseusing unstructured information as a source, the PDM step just returnsthe LAT itself. In type-and-generate, this step corresponds to producinga semantic answer type (SAT) from the question. PDM corresponds stronglyto notions of word sense disambiguation with respect to a specificsource.

Type Retrieval (TR):

After EDM, the types of the retrieved entity are retrieved. For someTyCor components, such as those using structured sources, this stepexercises the primary function of the source and is simple. In others,such as those using unstructured sources, this may require parsing orother semantic processing of some small snippet of natural language.

Type Alignment:

The results of the PDM and TR steps are then compared to determine thedegree of match. In sources containing e.g., a type taxonomy, thisincludes checking the taxonomy for subsumption, disjointness, etc. Forother sources, type alignments utilize resources such as WordNet forfinding synonyms, hypernyms, etc. between the types.

Each of the steps above generates a type-score reflecting the accuracyof its operation, taking into account the uncertainty of the entitymapping or information retrieval process. The final score produced byeach of the TyCor components 155 is a combination of the analysis stepscores and the confidence in the type of the candidate answer. Someexamples of specific scoring methodologies are described in detail inU.S. Patent Application Publication No. US 2009/0292687 A1.

In particular, the candidate answer 132 and type(s) are represented aslexical strings. Production of the score, referred to herein as thetype-score, is comprised of several steps: candidate answer to instancematching, instance to type association extraction, and LAT to typematching. The type-score reflects the degree to which the candidateanswer may be “coerced” to the LAT, where higher scores indicate abetter coercion.

In candidate answer to instance matching, the candidate answer ismatched against an instance or instances within the knowledge resource,where the form that the instance takes depends on the knowledgeresource. With a structured knowledge base, instances may be entities;with an encyclopedic source, such as Wikipedia, instances may be entriesin the encyclopedia; with lexical resources, such as WordNet (lexicaldatabase), instances may be synset entries (sets of synonyms); and withunstructured document (or webpage) collections, instances may be anyterms or phrases occurring within the text. If multiple instances arefound, a rollup using an aggregation function may be employed to combinethe scores from all candidate answers. If no suitable instance is found,a score of 0 is returned.

Next, instance association information is extracted from the resource.This information associates each instance with a type or set of types.Depending on the resource, this may take different forms; in a knowledgebase, this corresponds to particular relations of interest that relateinstances to types, with an encyclopedic source, this could be lexicalcategory information which assigns a lexical type to an entity, withlexical resources, such as WordNet, this is a set of lexical relations,such as hyponymy, over synsets (e.g., “artist” is a “person”), and withunstructured document collections this could be co-occurrence orproximity to other terms and phrases representing type.

Then, each LAT is attempted to match against each type. A lexicalmanifestation of the type is used. For example, with encyclopedias, thiscould be the string representing the category; with a lexical resource,such as WordNet, this could be the set of strings contained within thesynset. The matching is performed by using string matching or additionallexical resources, such as WordNet, to check for synonymy or hyponymybetween the LAT and type. Special logic may be implemented for types ofinterest; for example, person matcher logic may be activated, whichrequires not a strict match, synonym, or hyponym relation, but ratherthat both LAT and type are hyponyms of the term “person”. In this way,“he” and “painter”, for example, would be given a positive score eventhough they are not strictly synonyms or hyponyms. Finally, the set ofpairs of scores scoring the degree of match may be resolved to a singlefinal score via an aggregation function. The final score represents thedegree of confidence that the candidate answer is a type that matchesthe question LAT 207.

The disclosure herein describes a method and system of combiningmultiple TyCor components 155 to determine if the candidate answer 132is of the lexical type 215 of the question. FIG. 3 shows an example oftype coercion chaining according to systems and method herein. As shownin FIG. 3, a first set of TyCor components 155 a-155 f may be used incombination to provide input to a second set of TyCor components 355a-355 g. As illustrated in FIG. 3, the first set of TyCor components 155a-155 f includes six components and the second set of TyCor components355 a-355 g includes seven components. Other numbers of TyCor componentsmay be used. Each of the TyCor components 155 a-155 f evaluates thecandidate answer 132 according to its own methodology and provides apreliminary type and a type-score based on its own confidence that thecandidate answer 132 matches the question LAT 207. The thresholds usedin the various TyCor components 155 a-155 f and the weights of theirrespective rules may be determined based on empirical evaluation.

The “answer-type” classification process depends on the resource thatthe candidate answer 132 comes from. Different methods are applied todifferent resources (structured vs. unstructured, semantic vs. lexical,etc.) to find the answer-type of a candidate answer 132. In other words,the different resources may produce different preliminary types 313.

First, each of a first set of TyCor components (Wiki-intro 155 a,Wiki-List 155 b, Identity 155 c, Wiki-category 155 d, WordNet 155 e, andYago 155 f) attempts to obtain a type of the input candidate answer 132.Because different TyCor components 155 a-155 f use different resourcesand different algorithms, they may extract different preliminary types313 for the same candidate answer.

That is, the answer type of a given candidate answer may be found byapplying a different resource-specific methodology to each differentresource or database for its type coercion methodology. For example,different resources (Wiki-List, WordNet, Yago, etc.) that might producea preliminary type 313 for the candidate answer have differentanswer-type finding algorithms applied to them to find the answer type.Thus, use of different resource-specific type classificationmethodologies may result in the different resources producing differenttype categories for the same candidate answer, simply because thecandidate answer 132 is evaluated differently.

Additionally, each of the TyCor components 155 a-155 f generates atype-score reflecting the accuracy of its operation, taking into accountthe uncertainty of the entity mapping or information retrieval process.The different resource-specific answer-type classification methodologiesmay score the probability that an answer-type matches the question LATdifferently. In other words, one resource-specific methodology mightgive a 70% probability that the answer-type matches the question LAT,while another resource-specific methodology might give a 90% probabilitythat the same answer-type matches the question LAT. Of course, differentresource-specific methodologies may also come up with differentanswer-types.

According to systems and methods herein, the results of the differentTyCor component combinations can be posted directly as features to helpscoring the preliminary type 313 for the candidate answer 132. Theproblem is not so much that the candidate answer 132 is “correct” butrather, in this evaluation, that the candidate answer 132 addresses theright type of question. For example, an input query, to wit:

“Which 19th century US presidents were assassinated?”

should compute a lexical answer type (LAT) as “19th century USpresident” (but also as “US president” and “president”). Candidateanswers might include “Abraham Lincoln”, “James A. Garfield”, “WilliamMcKinley”, and “John F. Kennedy”, the latter two of which would be wronganswers but may be the correct type. Candidate answers that include“Secretariat” and “War of 1812” should be evaluated as being a wrongtype.

The output of each of the first set of TyCor components 155 a-155 fprovides input to each of another set of TyCor components (in this caseWiki-intro 355 a, Wiki-List 355 b, Identity 355 c, Wiki-category 355 d,WordNet 355 e, Yago 355 f, and Prismatic 355 g). Note that the sets ofTyCor components used need not be identical. According to systems andmethods herein, any combinations of TyCor components can be used. Theoutput from each of the first set of TyCor components 155 a-155 fincludes a preliminary type 313 and a first type-score.

The second set of TyCor components 355 a-355 g evaluates the preliminarytype 313 for the candidate answer and first type-scores from each of thefirst set of TyCor components 155 a-155 f in order to produce a secondscore for the preliminary type 313. The second score is based on acombination of the first type-score and a measure of degree that thepreliminary type 313 matches the question LAT 207 based on themethodology and resources for each of the second set of TyCor components355 a-355 g. The second set of TyCor components 355 a-355 g can use theprovenance information (i.e., which TyCor component extracted thepreliminary type 313 for the candidate answer) as features.

The classifier 373 evaluates the second score from each of the secondset of TyCor components 355 a-355 g and determines a final scorerepresenting the degree of confidence that the candidate answer 132 isof the lexical type of the question by aggregating the second score fromeach of the second set of TyCor components 355 a-355 g. The classifier373 can be trained from a set of ground truth instances of annotatedtraining data by machine learning algorithms, such as logisticregression, or can be implemented as a set of manually crafted rules.Classification rules can be encoded manually or learned automaticallythrough association rule learning.

Additionally, the second set of TyCor components 355 a-355 g can be usedto determine if the preliminary type 313 is subsumed by the question LAT207 “leader” (i.e., is a subtype of the question LAT 207). Note that thesets of TyCor components used are not identical. According to systemsand methods herein, any combinations of TyCor components can be used todetermine the possible types of the instances, and to determine whetherthe types are subtypes of the question LAT 207.

FIG. 4 shows an example of type coercion chaining. If the examplequestion is:

“What leader gave a speech at the dedication of a national cemetery inGettysburg, Pa.?”

Based on analysis, the question LAT 207 may be “leader”. A candidateanswer 432 may be “Lincoln”. The task is to determine if “Lincoln” is aleader. The chain of TyCor components may produce an output through thefollowing process:

First, each of a first set of TyCor components (Wiki-intro 155 a,Wiki-List 155 b, Identity 155 c, Wiki-category 155 d, WordNet 155 e, andYago 155 f) attempts to obtain a type of the input candidate answer 432.Because different TyCor components 155 a-155 f use different resourcesand different algorithms, they may extract different preliminary typesfor the same candidate answer. In this example, the candidate answer 432of “Lincoln” may extract two or more preliminary types, such as,President 443 and Politician 446. While a preliminary type of Automobilemay be extracted, it is expected that that would have a very low firstscore.

As described above, the preliminary type of a given candidate answer maybe found by applying a different resource-specific methodology to eachdifferent resource that produces a candidate answer. For example,different resources (Wiki-List, WordNet, Yago, etc.) that might producea candidate answer have different answer-type finding algorithms appliedto them to find the answer type. Thus, use of differentresource-specific type classification methodologies may result in thedifferent resources producing different type categories (presidentanswer-type and politician answer-type) for the same candidate answer,simply because the candidate answer 432, Lincoln, came from differentresources.

The output of the first set of TyCor components 155 a-155 f providesinput to each of a second set of TyCor components (Wiki-intro 355 a,Wiki-List 355 b, Identity 355 c, Wiki-category 355 d, WordNet 355 e,Yago 355 f, and Prismatic 355 g). The output from each of the first setof TyCor components 155 a-155 f includes a preliminary type, such asPresident 443 and Politician 446, and a first type-score. The second setof TyCor components 355 a-355 g determine if the types (e.g., President443 and Politician 446) are subsumed by the question LAT 207 “leader”(i.e., is a subtype of the question LAT 207).

As described above, each of the first set of TyCor components 155 a-155f generates a type-score reflecting the accuracy of its operation,taking into account the uncertainty of the entity mapping or informationretrieval process. The different resource-specific answer-typeclassification methodologies may score the probability that apreliminary type matches the question LAT 207 differently. In thisexample, one resource-specific methodology might give a 70% probabilitythat the preliminary type “president” 443 matches the “leader” LAT,while another resource-specific methodology might give a 90% probabilitythat the same preliminary type “president” 443 matches the “leader” LAT.Of course, different resource-specific methodologies will also come upwith different preliminary types (e.g., preliminary type “politician”446).

The second set of TyCor components 355 a-355 g evaluates the preliminarytypes President 443 and Politician 446 for the candidate answer and thefirst type-score from each of the first set of TyCor components 155a-155 f in order to produce a second score for each of the preliminarytypes President 443 and Politician 446. The second score is based on acombination of the first type-score and a measure of degree that thepreliminary types President 443 and Politician 446 match the questionLAT 207 “leader” based on the methodology and resources for each of thesecond set of TyCor components 355 a-355 g. The second score is anumeric value to indicate the degree of matchness of type-to-LAT.

The classifier 373 uses the outputs of the second set of TyCorcomponents 355 a-355 g to determine whether the candidate answer 432“Lincoln” is a leader or not. The classifier 373 evaluates the secondscore from each of the second set of TyCor components 355 a-355 g anddetermines a final score representing the confidence that the candidateanswer 432 “Lincoln” is of the lexical type of the question (“leader”).

The classifier 373 aggregates the second score from each of thecomponents of the second set of TyCor components 355 a-355 g and usesthe provenance information (i.e., which TyCor component extracted thetype for the candidate answer and which TyCor components determine thepreliminary type is a subtype of the question LAT) as features. Theoutput of the classifier 373 is a final score representing theconfidence that the candidate answer 432 “Lincoln” is of the lexicaltype (“leader”) of the question.

The principles of the systems and methods disclosed herein may beadvantageously employed in applications of several vertical domains:domains where applications involving text analytics are growing, andwhere navigation through large amounts of data is becoming of concern.For instance, example domains may include, but are not limited to, e.g.,biotech, finance, marketing, legal, etc.

Moreover, the present concepts may be extended to cover informationseeking dialog, not just question answering. This has application bothfor spoken dialog, e.g., between man and machine (telephone, computer)and, e.g., for multimodal dialog using speech and text that leveragesboth rule based approach to creating of dialog models or a combinationof statistical and rule based dialog modeling. As such, it does notpermit context switching without a large degradation of performance.Deferred type matching as performed in accordance with the system andmethods herein, should decrease degradation and allow context switching.

Yet in another system and method, the concepts may be extended to coverinformation mining where data is often of poor quality but contextshould help, e.g., information extraction from meeting recordings.

FIG. 5 is a flow diagram illustrating the processing flow of anexemplary method of determining a confidence score for candidate answersto questions in a question-answering system according to systems andmethods herein. At 509, a question is received into an automated QAsystem comprising a corpus of data. Automated query analysis isperformed, at 518, to determine a lexical answer type (LAT) associatedwith the question. At 527, a candidate answer generator identifies acandidate answer to the question using the corpus of data. At 536,preliminary types are determined for the candidate answer, using a firstplurality of TyCor components. Each of the first plurality of TyCorcomponents uses different resources (algorithms and/or databases) toproduce the preliminary types, and each of the first plurality of TyCorcomponents produces a preliminary type. The TyCor components may producedifferent preliminary types for a same candidate answer, based ondifferent methods. At 545, each of the first plurality of TyCorcomponents compares the preliminary type to the question LAT and, at554, using an automated scoring function of the TyCor components,produces first type-scores for the preliminary type, based on an amountthat the preliminary type corresponds to the question LAT. A samepreliminary type for the candidate answer may receive differenttype-scores based on differences between the TyCor components thatproduced the same preliminary type. At 563, each of the preliminarytypes for the candidate answer and the first type-scores are evaluatedusing a second plurality of TyCor components. Each of the secondplurality of TyCor components produces a second score for thepreliminary type, at 572, based on a combination of the firsttype-scores and a measure of degree that the preliminary type matchesthe question LAT. The scoring is differentiated based on which of thefirst plurality of TyCor components produced the preliminary types. Eachof the second plurality of TyCor components evaluates each preliminarytype and the first type-score from each one of the first plurality ofTyCor components. In some methods, the second plurality of TyCorcomponents determines whether the preliminary types are subtypes of thequestion LAT, at 581, based on how well the candidate answer types matchthe question LAT. At 590, a final score representing a degree ofconfidence that the candidate answer is a type that matches the questionLAT is output. The final score is based on an aggregation of the secondscore from each of the second plurality of TyCor components.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to various systemsand methods. It will be understood that each block of the flowchartillustrations and/or two-dimensional block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. The computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

According to a further system and method herein, an article ofmanufacture is provided that includes a tangible computer readablemedium having computer readable instructions embodied therein forperforming the steps of the computer implemented methods, including, butnot limited to, the method illustrated in FIG. 5. Any combination of oneor more computer readable non-transitory medium(s) may be utilized. Thecomputer readable medium may be a computer readable signal medium or acomputer readable storage medium. The non-transitory computer storagemedium stores instructions, and a processor executes the instructions toperform the methods described herein. A computer readable storage mediummay be, for example, but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,or device, or any suitable combination of the foregoing. Any of thesedevices may have computer readable instructions for carrying out thesteps of the methods described above with reference to FIG. 5.

The computer program instructions may be stored in a computer readablemedium that can direct a computer, other programmable data processingapparatus, or other devices to function in a particular manner, suchthat the instructions stored in the computer readable medium produce anarticle of manufacture including instructions which implement thefunction/act specified in the flowchart and/or block diagram block orblocks.

Furthermore, the computer program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other devicesto cause a series of operational steps to be performed on the computer,other programmable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

In case of implementing the systems and methods herein by softwareand/or firmware, a program constituting the software may be installedinto a computer with dedicated hardware, from a storage medium or anetwork, and the computer is capable of performing various functions ifwith various programs installed therein.

A representative hardware environment for practicing the systems andmethods herein is depicted in FIG. 6. This schematic drawing illustratesa hardware configuration of an information handling/computer system inaccordance with the systems and methods herein. The system comprises atleast one processor or central processing unit (CPU) 710. The CPUs 710are interconnected via system bus 712 to various devices such as aRandom Access Memory (RAM) 714, Read-Only Memory (ROM) 716, and anInput/Output (I/O) adapter 718. The I/O adapter 718 can connect toperipheral devices, such as disk units 711 and tape drives 713, or otherprogram storage devices that are readable by the system. The system canread the inventive instructions on the program storage devices andfollow these instructions to execute the methodology of the systems andmethods herein.

In FIG. 6, CPUs 710 perform various processing based on a program storedin a Read Only Memory (ROM) 716 or a program loaded from a peripheraldevice, such as disk units 711 and tape drives 713 to a Random AccessMemory (RAM) 714. In the RAM 714, required data when the CPU 710performs the various processing or the like is also stored, asnecessary. The CPU 710, the ROM 716, and the RAM 714 are connected toone another via a bus 712. An input/output adapter 718 is also connectedto the bus 712 to provide an input/output interface, as necessary. Aremovable medium, such as a magnetic disk, an optical disk, amagneto-optical disk, a semiconductor memory, or the like, is installedon the peripheral device, as necessary, so that a computer program readtherefrom may be installed into the RAM 714, as necessary.

The system further includes a user interface adapter 719 that connects akeyboard 715, mouse 717, speaker 724, microphone 722, and/or other userinterface devices such as a touch screen device (not shown) to the bus712 to gather user input. Additionally, a communication adapter 720including a network interface card such as a LAN card, a modem, or thelike connects the bus 712 to a data processing network 725. Thecommunication adapter 720 performs communication processing via anetwork such as the Internet. A display adapter 721 connects the bus 712to a display device 723, which may be embodied as an output device suchas a monitor (such as a Cathode Ray Tube (CRT), a Liquid Crystal Display(LCD), or the like), printer, or transmitter, for example.

In the case where the above-described series of processing isimplemented with software, the program that constitutes the software maybe installed from a network such as the Internet or a storage mediumsuch as the removable medium.

Those skilled in the art would appreciate that the storage medium is notlimited to the peripheral device having the program stored therein asillustrated in FIG. 6, which is distributed separately from the devicefor providing the program to the user. Examples of a removable mediuminclude a magnetic disk (including a floppy disk), an optical disk(including a Compact Disk-Read Only Memory (CD-ROM) and a DigitalVersatile Disk (DVD)), a magneto-optical disk (including a Mini-Disk(MD) (registered trademark)), and a semiconductor memory. Alternatively,the storage medium may be the ROM 716, a hard disk contained in thestorage section 711, or the like, which has the program stored thereinand is distributed to the user together with the device that containsthem.

As will be appreciated by one skilled in the art, aspects of the systemsand methods herein may be embodied as a system, method, or computerprogram product. Accordingly, aspects of the present disclosure may takethe form of an entirely hardware system, an entirely software system(including firmware, resident software, micro-code, etc.) or an systemcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module”, or “system.” Furthermore,aspects of the present disclosure may take the form of a computerprogram product embodied in one or more computer readable medium(s)having computer readable program code embodied thereon.

Any combination of one or more computer readable non-transitorymedium(s) may be utilized. The computer readable medium may be acomputer readable signal medium or a computer readable storage medium.The non-transitory computer storage medium stores instructions, and aprocessor executes the instructions to perform the methods describedherein. A computer readable storage medium may be, for example, but notlimited to, an electronic, magnetic, optical, electromagnetic, infrared,or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a Read-Only Memory(ROM), an Erasable Programmable Read-Only Memory (EPROM or Flashmemory), an optical fiber, a magnetic storage device, a portable compactdisc Read-Only Memory (CD-ROM), an optical storage device, a“plug-and-play” memory device, like a USB flash drive, or any suitablecombination of the foregoing. In the context of this document, acomputer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including, but not limited to, wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++, or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer, or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to varioussystems and methods herein. In this regard, each block in the flowchartor block diagrams may represent a module, segment, or portion of code,which comprises one or more executable instructions for implementing thespecified logical function(s). It should also be noted that, in somealternative implementations, the functions noted in the block mightoccur out of the order noted in the Figures. For example, two blocksshown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Deployment types include loading directly in the client, server, andproxy computers via loading a storage medium such as a CD, DVD, etc. Theprocess software may also be automatically or semi-automaticallydeployed into a computer system by sending the process software to acentral server or a group of central servers. The process software isthen downloaded into the client computers that will execute the processsoftware. The process software is sent directly to the client system viae-mail. The process software is then either detached to a directory orloaded into a directory by a button on the e-mail that executes aprogram that detaches the process software into a directory.Alternatively, the process software is sent directly to a directory onthe client computer hard drive. When there are proxy servers, theprocess will select the proxy server code, determine on which computersto place the proxy servers' code, transmit the proxy server code, andthen install the proxy server code on the proxy computer. The processsoftware will be transmitted to the proxy server, and then stored on theproxy server.

While it is understood that the process software may be deployed bymanually loading directly in the client, server, and proxy computers vialoading a storage medium such as a CD, DVD, etc., the process softwaremay also be automatically or semi-automatically deployed into a computersystem by sending the process software to a central server or a group ofcentral servers. The process software is then downloaded into the clientcomputers that will execute the process software. Alternatively, theprocess software is sent directly to the client system via e-mail. Theprocess software is then either detached to a directory or loaded into adirectory by a button on the e-mail that executes a program thatdetaches the process software into a directory. Another alternative isto send the process software directly to a directory on the clientcomputer hard drive. When there are proxy servers, the process willselect the proxy server code, determine on which computers to place theproxy servers' code, transmit the proxy server code, and then installthe proxy server code on the proxy computer. The process software willbe transmitted to the proxy server, and then stored on the proxy server.

In FIG. 7, step 800 begins the deployment of the process software. Thefirst thing is to determine if there are any programs that will resideon a server or servers when the process software is executed 801. Ifthis is the case, then the servers that will contain the executables areidentified 909. The process software for the server or servers istransferred directly to the servers' storage via FTP or some otherprotocol, or by copying through the use of a shared file system 910. Theprocess software is then installed on the servers 911.

Next, a determination is made on whether the process software is to bedeployed by having users access the process software on a server orservers 802. If the users are to access the process software on servers,then the server addresses that will store the process software areidentified 803.

A determination is made if a proxy server is to be built 900 to storethe process software. A proxy server is a server that sits between aclient application, such as a Web browser, and a real server. Itintercepts all requests to the real server to see if it can fulfill therequests itself. If not, it forwards the request to the real server. Thetwo primary benefits of a proxy server are to improve performance and tofilter requests. If a proxy server is required, then the proxy server isinstalled 901. The process software is either sent to the servers via aprotocol such as FTP or it is copied directly from the source files tothe server files via file sharing 902. Another method would be to send atransaction to the servers that contain the process software and havethe server process the transaction, then receive and copy the processsoftware to the server's file system. Once the process software isstored at the servers, the users, via their client computers, thenaccess the process software on the servers and copy it to their clientcomputers file systems 903. Another method is to have the serversautomatically copy the process software to each client and then run theinstallation program for the process software at each client computer.The users execute the program that installs the process software ontheir client computer 912, and then exit the process 808.

In step 804, a determination is made whether the process software is tobe deployed by sending the process software to users via e-mail. The setof users where the process software will be deployed are identifiedtogether with the addresses of the user client computers 805. Theprocess software is sent via e-mail 904 to each of the users' clientcomputers. The users receive the e-mail 905 and then detach the processsoftware from the e-mail to a directory on their client computers 906.The users execute the program that installs the process software ontheir client computer 912, and then exit the process 808.

Lastly, a determination is made on whether the process software will besent directly to user directories on their client computers 806. If so,the user directories are identified 807. The process software istransferred directly to the users' client computer directory 907. Thiscan be done in several ways such as, but not limited to, sharing of thefile system directories and then copying from the sender's file systemto the recipient users' file system or alternatively using a transferprotocol such as File Transfer Protocol (FTP). The users access thedirectories on their client file systems in preparation for installingthe process software 908. The users execute the program that installsthe process software on their client computer 912, and then exit theprocess 808.

The process software is integrated into a client, server, and networkenvironment by providing for the process software to coexist withapplications, operating systems, and network operating systems software,and then installing the process software on the clients and servers inthe environment where the process software will function.

The first step is to identify any software on the clients and serversincluding the network operating system where the process software willbe deployed that are required by the process software or that work inconjunction with the process software. This includes the networkoperating system that is software that enhances a basic operating systemby adding networking features.

Next, the software applications and version numbers will be identifiedand compared to the list of software applications and version numbersthat have been tested to work with the process software. Those softwareapplications that are missing or that do not match the correct versionwill be upgraded with the correct version numbers. Program instructionsthat pass parameters from the process software to the softwareapplications will be checked to ensure the parameter lists match theparameter lists required by the process software. Conversely, parameterspassed by the software applications to the process software will bechecked to ensure the parameters match the parameters required by theprocess software. The client and server operating systems including thenetwork operating systems will be identified and compared to the list ofoperating systems, version numbers, and network software that have beentested to work with the process software. Those operating systems,version numbers, and network software that do not match the list oftested operating systems and version numbers will be upgraded on theclients and servers to the required level.

After ensuring that the software, where the process software is to bedeployed, is at the correct version level that has been tested to workwith the process software, the integration is completed by installingthe process software on the clients and servers.

In FIG. 8, step 920 begins the integration of the process software. Thefirst thing is to determine if there are any process software programsthat will execute on a server or servers 921. If this is not the case,then integration proceeds to 927. If this is the case, then the serveraddresses are identified 922. The servers are checked to see if theycontain software that includes the operating system (OS), applications,and network operating systems (NOS), together with their version numbersthat have been tested with the process software 923. The servers arealso checked to determine if there is any missing software that isrequired by the process software 923.

A determination is made if the version numbers match the version numbersof OS, applications, and NOS that have been tested with the processsoftware 924. If all of the versions match and there is no missingrequired software, the integration continues in 927.

If one or more of the version numbers do not match, then the unmatchedversions are updated on the server or servers with the correct versions925. Additionally, if there is missing required software, then it isupdated on the server or servers 925. The server integration iscompleted by installing the process software 926.

Step 927, which follows either step 921, 924, or 926, determines ifthere are any programs of the process software that will execute on theclients. If no process software programs execute on the clients, theintegration proceeds to 930 and exits. If this not the case, then theclient addresses are identified 928.

The clients are checked to see if they contain software that includesthe operating system (OS), applications, and network operating systems(NOS), together with their version numbers that have been tested withthe process software 929. The clients are also checked to determine ifthere is any missing software that is required by the process software929.

A determination is made as to whether the version numbers match theversion numbers of OS, applications, and NOS that have been tested withthe process software 931. If all of the versions match and there is nomissing required software, then the integration proceeds to 930 andexits.

If one or more of the version numbers do not match, then the unmatchedversions are updated on the clients with the correct versions 932. Inaddition, if there is missing required software then it is updated onthe clients 932. The client integration is completed by installing theprocess software on the clients 933. The integration proceeds to 930 andexits.

The process software can be stored on a shared file system accessiblefrom one or more servers. The process software is executed viatransactions that contain data and server processing requests that useCPU units on the accessed server. CPU units are units of time such asminutes, seconds, and hours on the central processor of the server.Additionally, the assessed server may make requests of other serversthat require CPU units. CPU units are an example that represents onemeasurement of use. Other measurements of use include, but are notlimited to, network bandwidth, memory usage, storage usage, packettransfers, complete transactions etc. When multiple customers use thesame process software application, their transactions are differentiatedby the parameters included in the transactions that identify the uniquecustomer and the type of service for that customer. All of the CPU unitsand other measurements of use that are used for the services for eachcustomer are recorded. When the number of transactions to any one serverreaches a number that begins to affect the performance of that server,other servers are accessed to increase the capacity and to share theworkload. Likewise, when other measurements of use such as networkbandwidth, memory usage, storage usage, etc. approach a capacity so asto affect performance, additional network bandwidth, memory usage,storage etc. are added to share the workload. The measurements of useused for each service and customer are sent to a collecting server thatsums the measurements of use for each customer for each service that wasprocessed anywhere in the network of servers that provide the sharedexecution of the process software. The summed measurements of use unitsare periodically multiplied by unit costs and the resulting totalprocess software application service costs are alternatively sent to thecustomer and/or indicated on a web site accessed by the customer, whichthen remits payment to the service provider. In another method, theservice provider requests payment directly from a customer account at abanking or financial institution. In another method, if the serviceprovider is also a customer of the customer that uses the processsoftware application, the payment owed to the service provider isreconciled to the payment owed by the service provider to minimize thetransfer of payments.

The process software is shared, simultaneously serving multiplecustomers in a flexible, automated fashion. It is standardized,requiring little customization, and it is scalable, providing capacityon demand in a pay-as-you-go model.

The process software can be stored on a shared file system accessiblefrom one or more servers. The process software is executed viatransactions that contain data and server processing requests that useCPU units on the accessed server. CPU units are units of time such asminutes, seconds, and hours on the central processor of the server.Additionally, the assessed server may make requests of other serversthat require CPU units. CPU units are an example that represents onemeasurement of use. Other measurements of use include, but are notlimited to, network bandwidth, memory usage, storage usage, packettransfers, complete transactions etc.

When multiple customers use the same process software application, theirtransactions are differentiated by the parameters included in thetransactions that identify the unique customer and the type of servicefor that customer. All of the CPU units and other measurements of usethat are used for the services for each customer are recorded. When thenumber of transactions to any one server reaches a number that begins toaffect the performance of that server, other servers are accessed toincrease the capacity and to share the workload. Likewise, when othermeasurements of use such as network bandwidth, memory usage, storageusage, etc. approach a capacity so as to affect performance, additionalnetwork bandwidth, memory usage, storage etc. are added to share theworkload.

The measurements of use used for each service and customer are sent to acollecting server that sums the measurements of use for each customerfor each service that was processed anywhere in the network of serversthat provide the shared execution of the process software. The summedmeasurements of use units are periodically multiplied by unit costs andthe resulting total process software application service costs arealternatively sent to the customer and/or indicated on a web siteaccessed by the customer, which then remits payment to the serviceprovider.

In another method, the service provider requests payment directly from acustomer account at a banking or financial institution.

In another method, if the service provider is also a customer of thecustomer that uses the process software application, the payment owed tothe service provider is reconciled to the payment owed by the serviceprovider to minimize the transfer of payments.

In FIG. 9, step 940 begins the On Demand process. A transaction iscreated that contains the unique customer identification, the requestedservice type, and any service parameters that further specify the typeof service 941. The transaction is then sent to the main server 942. Inan On Demand environment, the main server can initially be the onlyserver, then, as capacity is consumed, other servers are added to the OnDemand environment.

The server central processing unit (CPU) capacities in the On Demandenvironment are queried 943. The CPU requirement of the transaction isestimated, then the servers' available CPU capacity in the On Demandenvironment are compared to the transaction CPU requirement to see ifthere is sufficient CPU capacity available in any server to process thetransaction 944. If there is not sufficient server CPU capacityavailable, then additional server CPU capacity is allocated to processthe transaction 948. If there was already sufficient CPU capacityavailable, then the transaction is sent to a selected server 945.

Before executing the transaction, a check is made of the remaining OnDemand environment to determine if the environment has sufficientavailable capacity for processing the transaction. This environmentcapacity consists of such things as, but not limited to, networkbandwidth, processor memory, storage etc. 946. If there is notsufficient available capacity, then capacity will be added to the OnDemand environment 947. Next, the required software to process thetransaction is accessed, loaded into memory, and then the transaction isexecuted 949.

The usage measurements are recorded 950. The usage measurements consistof the portions of those functions in the On Demand environment that areused to process the transaction. The usage of such functions as, but notlimited to, network bandwidth, processor memory, storage, and CPU cyclesare what is recorded. The usage measurements are summed, multiplied byunit costs, and then recorded as a charge to the requesting customer951. If the customer has requested that the On Demand costs be posted toa web site 952, then they are posted 953.

If the customer has requested that the On Demand costs be sent viae-mail to a customer address 954, then they are sent 955. If thecustomer has requested that the On Demand costs be paid directly from acustomer account 956, then payment is received directly from thecustomer account 957. The last step is to exit the On Demand process958.

The process software may be deployed, accessed and executed through theuse of a virtual private network (VPN), which is any combination oftechnologies that can be used to secure a connection through anotherwise unsecured or untrusted network. The use of VPNs is to improvesecurity and for reduced operational costs. The VPN makes use of apublic network, usually the Internet, to connect remote sites or userstogether. Instead of using a dedicated, real-world connection such asleased line, the VPN uses “virtual” connections routed through theInternet from the company's private network to the remote site oremployee.

The process software may be deployed, accessed, and executed througheither a remote-access or a site-to-site VPN. When using theremote-access VPNs, the process software is deployed, accessed, andexecuted via the secure, encrypted connections between a company'sprivate network and remote users through a third-party service provider.The enterprise service provider (ESP) sets a network access server (NAS)and provides the remote users with desktop client software for theircomputers. The telecommuters can then dial a toll-free number or attachdirectly via a cable or DSL modem to reach the NAS and use their VPNclient software to access the corporate network and to access, download,and execute the process software.

When using the site-to-site VPN, the process software is deployed,accessed, and executed through the use of dedicated equipment andlarge-scale encryption, which are used to connect a company's multiplefixed sites over a public network, such as the Internet.

The process software is transported over the VPN via tunneling, which isthe process of placing an entire packet within another packet andsending it over a network. The protocol of the outer packet isunderstood by the network and both points, called tunnel interfaces,where the packet enters and exits the network.

In FIGS. 10 and 11, step 960 begins the Virtual Private Network (VPN)process. A determination is made to see if a VPN for remote access isrequired 961. If it is not required, then proceed to 962. If it isrequired, then determine if the remote access VPN exists 964.

If it does exist, then proceed to 965. Otherwise, identify the thirdparty provider that will provide the secure, encrypted connectionsbetween the company's private network and the company's remote users976. The company's remote users are identified 977. The third partyprovider then sets up a network access server (NAS) 978 that allows theremote users to dial a toll-free number or attach directly via a cableor DSL modem to access, download, and install the desktop clientsoftware for the remote-access VPN 979.

After the remote access VPN has been built, or if it been previouslyinstalled, the remote users can then access the process software bydialing into the NAS or attaching directly via a cable or DSL modem intothe NAS 965. This allows entry into the corporate network where theprocess software is accessed 966. The process software is transported tothe remote users' desktop over the network via tunneling. That is, theprocess software is divided into packets and each packet, including thedata and protocol, is placed within another packet 967. When the processsoftware arrives at the remote users' desktop, it is removed from thepackets, reconstituted, and executed on the remote users' desktop 968.

A determination is made to see if a VPN for site-to-site access isrequired 962. If it is not required, then proceed to exit the process963. Otherwise, determine if the site-to-site VPN exists 969. If it doesexist, then proceed to 972. Otherwise, install the dedicated equipmentrequired to establish a site-to-site VPN 970. Then build the large-scaleencryption into the VPN 971.

After the site-to-site VPN has been built, or if it had been previouslyestablished, the users access the process software via the VPN 972. Theprocess software is transported to the site users over the network viatunneling 973. That is, the process software is divided into packets andeach packet, including the data and protocol, is placed within anotherpacket 974. When the process software arrives at the remote users'desktop, it is removed from the packets, reconstituted, and executed onthe site users' desktop 975. Proceed to exit the process 963.

The terminology used herein is for the purpose of describing particularexamples of the disclosed structures and methods and is not intended tobe limiting of this disclosure. For example, as used herein, thesingular forms “a”, “an”, and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise.Additionally, as used herein, the terms “comprises,” “comprising,”and/or “including,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. Further, the terms “automated” or“automatically” mean that once a process is started (by a machine or auser), one or more machines perform the process without further inputfrom any user.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescriptions of the various systems and methods herein have beenpresented for purposes of illustration, but are not intended to beexhaustive or limited to the systems and methods disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedsystems and methods. The terminology used herein was chosen to bestexplain the principles of the systems and methods, the practicalapplication or technical improvement over technologies found in themarketplace, or to enable others of ordinary skill in the art tounderstand the systems and methods disclosed herein.

What is claimed is:
 1. A method comprising: receiving a question into a computerized device; determining a lexical answer type associated with said question, using said computerized device; identifying candidate answers to said question, using said computerized device; determining preliminary types for each of said candidate answers, using said computerized device; producing first type-scores, said first type-scores representing the degree of match between said preliminary types and said lexical answer type associated with said question, using a plurality of first scoring methods of said computerized device; producing second type-scores, said second type-scores representing a combination of said first type-score and the degree of match between said preliminary types and said lexical answer type associated with said question, using a plurality of second scoring methods of said computerized device; calculating a final score based on said second type-scores for each of said candidate answers, using said computerized device; and outputting said final score representing a degree of confidence that each of said candidate answers is a type that matches said lexical answer type associated with said question, using said computerized device.
 2. The method according to claim 1, further comprising: performing automated query analysis to determine said lexical answer type associated with said question, using said computerized device.
 3. The method according to claim 1, further comprising: matching said candidate answers against instances in a corpus of data, using said computerized device; retrieving preliminary types from said corpus of data, using said computerized device, said preliminary types being associated with said instances; matching said lexical answer type associated with said question with said preliminary types retrieved from said corpus of data, using said computerized device; and producing scores representing a degree of match between said lexical answer type associated with said question and said preliminary types for said candidate answers, using said computerized device.
 4. The method according to claim 1, said producing first type-scores comprising using first type coercion components of said computerized device, each of said first type coercion components having different resource-specific type classification methodologies scoring a measure of degree that said preliminary types match said lexical answer type associated with said question, each of said first type coercion components producing a preliminary type and a first type-score.
 5. The method according to claim 1, said producing second type-scores comprising using second type coercion components of said computerized device, each of said second type coercion components having different resource-specific type classification methodologies scoring a measure of degree that said preliminary types match said lexical answer type associated with said question, each of said second type coercion components producing a second type-score.
 6. The method according to claim 1, said calculating a final score based on said second type-scores comprising using an aggregation function resolving said second type-scores from each scoring method of said plurality of second scoring methods to a single final score, using said computerized device.
 7. The method according to claim 1, further comprising determining whether any of said preliminary types are subtypes of said lexical answer type associated with said question, using said computerized device.
 8. A method comprising: identifying a question lexical answer type (LAT) for a question in a question-answering system, using a computerized device; generating candidate answers to said question, using said computerized device; determining preliminary types for each of said candidate answers using different resources to produce said preliminary types, using said computerized device; scoring a match between said preliminary types and said question LAT using a plurality of first type coercion components of said computerized device, each of said first type coercion components producing a first type-score representing the degree of match between said preliminary types and said question LAT, said first type coercion components using different resources to produce said first type-score, and said scoring being differentiated based on which of said first type coercion components produced said a given preliminary type; evaluating each said preliminary type and each said first type-score from each of said first type coercion components, using a plurality of second type coercion components of said computerized device, each of said second type coercion components producing a second type-score for said preliminary type for said candidate answer based on a combination of said first type-score and a measure of degree that said preliminary type matches said question LAT; and outputting a final score based on said second type-score from each of said second type coercion components, said final score representing a degree of confidence that said candidate answer is a type that matches said question LAT, using said computerized device.
 9. The method according to claim 8, further comprising: receiving a question into said computerized device; and performing automated query analysis to determine said question LAT, using said computerized device.
 10. The method according to claim 8, said question-answering system further comprising a corpus of data, said method further comprising: generating said candidate answers for said question from said corpus of data, using said computerized device.
 11. The method according to claim 10, said scoring further comprising: matching said candidate answers against instances in said corpus of data, using said computerized device; retrieving preliminary types from said corpus of data, using said computerized device, said preliminary types being associated with said instances; matching said question LAT with said preliminary types retrieved from said corpus of data, using said computerized device; and producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answer, using said computerized device.
 12. The method according to claim 8, said first type coercion components having different resource-specific type classification methodologies scoring said measure of degree that said preliminary types match said question LAT.
 13. The method of claim 8, said second type coercion components having different resource-specific type classification methodologies scoring said measure of degree that said preliminary types match said question LAT.
 14. A computer program product for determining a confidence score for candidate answers to questions in a question-answering system, said computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, said program code being readable and executable by a computer to perform a method comprising: receiving a question into a computerized device; determining a question lexical answer type (LAT) associated with said question; identifying candidate answers to said question; determining preliminary types for each of said candidate answers; producing first type-scores representing the degree of match between said preliminary types and said question LAT, using a plurality of first scoring methods; producing second type-scores representing a combination of said first type-score and the degree of match between said preliminary types and said question LAT, using a plurality of second scoring methods; calculating a final score based on said second type-scores from each of said candidate answers; and outputting said final score representing a degree of confidence that said candidate answer is a type that matches said question LAT.
 15. The computer program product according to claim 14, said method further comprising: performing automated query analysis to determine said question LAT.
 16. The computer program product according to claim 14, said method further comprising: matching said candidate answer against instances in a corpus of data; retrieving preliminary types from said corpus of data, said preliminary types being associated with said instances; matching said question LAT with said preliminary types retrieved from said corpus of data; and producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answers.
 17. The computer program product according to claim 14, said producing first type-scores comprising using first type coercion components of said computerized device, each of said first type coercion components having different resource-specific type classification methodologies scoring a measure of degree that said preliminary types match said question LAT, each of said first type coercion components producing a preliminary type and a first type-score.
 18. The computer program product according to claim 14, said producing second type-scores comprising using second type coercion components of said computerized device, each of said second type coercion components having different resource-specific type classification methodologies scoring a measure of degree that said preliminary types match said question LAT, each of said second type coercion components producing a second type-score.
 19. The computer program product according to claim 14, said calculating a final score based on said second type-scores comprising using an aggregation function resolving said second type-scores from each scoring method of said plurality of second scoring methods to a single final score, using said computerized device.
 20. The computer program product according to claim 14, further comprising determining whether any of said preliminary types are subtypes of said question LAT, using said computerized device. 